T7101: Add hardware watchdog support via systemd #4843

Firefishy · 2025-11-10T15:19:57Z

Change summary

Add vyos support for having the hardware watchdog managed by systemd. Have systemd also use the hardware watchdog to control shutdown and reboot timeouts.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Code style update (formatting, renaming)
Refactoring (no functional changes)
Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
Other (please describe):

Related Task(s)

https://vyos.dev/T7101

Related PR(s)

vyos/vyos-documentation#1703 (feature documentation)

How to test / Smoketest result

Checklist:

I have read the CONTRIBUTING document
I have linked this PR to one or more Phabricator Task(s)
I have run the components SMOKETESTS if applicable
My commit headlines contain a valid Task id
My change requires a change to the documentation
I have updated the documentation accordingly

github-actions · 2025-11-10T15:20:18Z

👍
No issues in PR Title / Commit Title

src/conf_mode/system_watchdog.py

Copilot

Pull Request Overview

This PR adds hardware watchdog support to VyOS, allowing users to configure systemd's runtime watchdog functionality to monitor system health and trigger automatic reboots on hang/freeze conditions.

Key changes:

New Python configuration module for hardware watchdog with configurable timeouts
XML interface definition for CLI commands to enable/disable watchdog and set parameters
Jinja2 template for systemd watchdog configuration

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/conf_mode/system_watchdog.py	Core configuration logic for watchdog including validation, module loading, and systemd configuration
interface-definitions/system_watchdog.xml.in	CLI interface definition with timeout parameters and module specification
data/templates/system/watchdog.conf.j2	Systemd configuration template for watchdog runtime settings

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/conf_mode/system_watchdog.py

Firefishy · 2025-11-10T21:18:36Z

Added a basic smoketest for the watchdog: smoketest/scripts/cli/test_system_watchdog.py

smoketest/scripts/cli/test_system_watchdog.py

Extra imports were removed

src/conf_mode/system_watchdog.py

Copilot

Pull Request Overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/conf_mode/system_watchdog.py

interface-definitions/system_watchdog.xml.in

Copilot

Pull Request Overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/conf_mode/system_watchdog.py

sever-sever · 2025-11-11T14:16:10Z

Binary cannot build
Try build the package dpkg-builpackage -us -uc -b in the vyos-build container

 <string>:0:0:ERROR:RELAXNGV:RELAXNG_ERR_INTEREXTRA: Extra element children in interleave
build/interface-definitions/system_watchdog.xml:3:0:ERROR:RELAXNGV:RELAXNG_ERR_CONTENTVALID: Element node failed to validate content
Interface definition file build/interface-definitions/system_watchdog.xml does not match the schema!
make[2]: *** [Makefile:32: interface_definitions] Error 1
make[2]: Leaving directory '/__w/vyos-1x/vyos-1x/packages/vyos-1x'
make[1]: *** [debian/rules:49: override_dh_auto_build] Error 2
make[1]: Leaving directory '/__w/vyos-1x/vyos-1x/packages/vyos-1x'
make: *** [debian/rules:39: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

interface-definitions/system_watchdog.xml.in

Firefishy · 2025-11-11T17:09:48Z

The failed smoktest looks like a false positive:

DEBUG - test_bgp_99_bmp (__main__.TestProtocolsBGP.test_bgp_99_bmp) ... FAIL
DEBUG - 
DEBUG - ======================================================================
DEBUG - FAIL: test_bgp_99_bmp (__main__.TestProtocolsBGP.test_bgp_99_bmp)
DEBUG - ----------------------------------------------------------------------
DEBUG - Traceback (most recent call last):
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py", line 231, in tearDown
DEBUG -     super().tearDown()
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 83, in tearDown
DEBUG -     self.assertEqual(self.mgmt_daemon_pid, process_named_running(mgmt_daemon))
DEBUG - AssertionError: 1664 != 98762
DEBUG - 
DEBUG - ----------------------------------------------------------------------
DEBUG - Ran 30 tests in 192.167s
DEBUG - 
DEBUG - FAILED (failures=1)

interface-definitions/system_watchdog.xml.in

Firefishy · 2025-11-12T17:17:43Z

I've made all the requested changes. Let me know if anything more is required.

A minor addition if desired might be to allow specify which /dev/watchdogX device to use. There can be multiple watchdog devices on some systems and it can sometimes be preferential to use a specific device.

Copilot

Pull Request Overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

smoketest/scripts/cli/test_system_watchdog.py

interface-definitions/system_watchdog.xml.in

Copilot · 2025-11-12T17:40:12Z

src/conf_mode/system_watchdog.py

+        try:
+            modules_load_directory.mkdir(exist_ok=True)
+            modules_load_file.write_text(f"{module}\n")
+        except Exception as e:
+            print(f"Warning: Failed writing modules-load configuration: {e}")


Using bare Exception in except clauses is too broad and could mask unexpected errors. Consider catching more specific exceptions (e.g., OSError, IOError) or letting unexpected exceptions propagate to be caught by the airbag error handler.

I'll leave this to VyOS team to decide.

Copilot · 2025-11-12T17:40:12Z

src/conf_mode/system_watchdog.py

+            try:
+                call(f'modprobe {module}')
+            except Exception as e:
+                print(f"Warning: Could not load watchdog module '{module}': {e}")


Using bare Exception in except clauses is too broad and could mask unexpected errors. Consider catching more specific exceptions related to process execution (e.g., OSError) or letting unexpected exceptions propagate to be caught by the airbag error handler.

I'll leave this to VyOS team to decide.

github-actions · 2025-11-12T19:46:53Z

CI integration ❌ failed!

Details

CI logs

CLI Smoketests (no interfaces) 👍 passed
CLI Smoketests VPP 👍 passed
CLI Smoketests (interfaces only) 👍 passed
Config tests 👍 passed
Config tests VPP 👍 passed
RAID1 tests ❌ failed
TPM tests 👍 passed

dmbaturin

I like the idea and the CLI seems fine to me. I left a couple of questions/suggestions that I'd like to address before we merge it, in addition to concerns people already pointed out.

dmbaturin · 2025-11-13T15:18:02Z

interface-definitions/system_watchdog.xml.in

+              <help>Kernel module to load for watchdog device (optional)</help>
+              <valueHelp>
+                <format>txt</format>
+                <description>Module name (e.g. 'softdog', 'iTCO_wdt', 'sp5100_tco')</description>


We generally prefer set-time validation over commit-time validation whenever possible, since it provides a faster feedback and the user doesn't have to commit to discover that some options are wrong. Two questions from this perspective:

How likely is it that the module for the hardware watchdog actually present in the system will not be loaded? Can we fully automate that and either load the module for the watchdog that the system has or fail the commit if there's no module matching the watchdog?

If we can't fully automate it, can we add an explicit list of available watchdog module for set-time validation?

dmbaturin · 2025-11-13T15:19:09Z

interface-definitions/system_watchdog.xml.in

+          </leafNode>
+          <leafNode name="timeout">
+            <properties>
+              <help>Watchdog timeout for runtime in seconds (1-86400)</help>


Is 86400 the upper limit for those timeouts in the kernel or systemd? If it's not, I believe we shouldn't introduce an artificial limit — one person's constant is another person's variable, as Alan Perlis quipped.

The max value of watchdog timeout varies per watchdog. Eg: bcm2835_wdt has a max timeout of 16 seconds. The iTCO_wdt module on v1 hardware is around 30 seconds and as high as 10mins on V2 hardware.

I will investigate further, it is likely that the max value can be read from some modules once it has been loaded.

sarthurdev

Thanks for quickly addressing the requested changes.

I think we just need to figure out how to handle the module validation, XML-based is always the preferred method. But we also should validate the specified module is a watchdog driver.

sarthurdev · 2025-11-13T22:47:15Z

src/conf_mode/system_watchdog.py

+watchdog_config_dir = Path('/run/systemd/system.conf.d')
+watchdog_config_file = Path(watchdog_config_dir / 'watchdog.conf')
+modules_load_directory = Path('/run/modules-load.d')
+modules_load_file = Path(modules_load_directory / 'watchdog.conf')


Suggested change

watchdog_config_dir = Path('/run/systemd/system.conf.d')

watchdog_config_file = Path(watchdog_config_dir / 'watchdog.conf')

modules_load_directory = Path('/run/modules-load.d')

modules_load_file = Path(modules_load_directory / 'watchdog.conf')

watchdog_config_dir = Path('/run/systemd/system.conf.d')

watchdog_config_file = watchdog_config_dir / 'watchdog.conf'

modules_load_directory = Path('/run/modules-load.d')

modules_load_file = modules_load_directory / 'watchdog.conf'

Don't need to encapsulate the file paths in another Path()

sarthurdev · 2025-11-13T22:51:50Z

src/conf_mode/system_watchdog.py

+    # If a module is provided, verify it resolves via modprobe (dry-run)
+    if module:
+        # Dry-run modprobe (-n) in quiet mode (-q) verifies availability without loading
+        rc = run(f'modprobe -n -q {module}')


Only issue here is that it doesn't verify the specified module is a valid watchdog driver.

As per @dmbaturin's comment, probably best to build a validator or list of watchdog modules enabled in the VyOS kernel.

github-actions bot assigned Firefishy Nov 10, 2025

github-actions bot added the current label Nov 10, 2025

Firefishy changed the title ~~T7101~~ T7101: Add hardware watchdog support via systemd Nov 10, 2025

Firefishy commented Nov 10, 2025

View reviewed changes

src/conf_mode/system_watchdog.py Outdated Show resolved Hide resolved

Firefishy force-pushed the T7101 branch from f97487a to 140ec61 Compare November 10, 2025 15:29

sever-sever requested a review from Copilot November 10, 2025 17:55

Copilot AI reviewed Nov 10, 2025

View reviewed changes

src/conf_mode/system_watchdog.py Outdated Show resolved Hide resolved

src/conf_mode/system_watchdog.py Outdated Show resolved Hide resolved

src/conf_mode/system_watchdog.py Outdated Show resolved Hide resolved

Firefishy force-pushed the T7101 branch 3 times, most recently from b066fe4 to 8796ec3 Compare November 10, 2025 21:18

Firefishy force-pushed the T7101 branch from 8796ec3 to 4a5ff9b Compare November 10, 2025 21:20

sever-sever previously requested changes Nov 11, 2025

View reviewed changes

smoketest/scripts/cli/test_system_watchdog.py Outdated Show resolved Hide resolved

sever-sever requested review from c-po, dmbaturin and sarthurdev November 11, 2025 02:29

Firefishy force-pushed the T7101 branch from 4a5ff9b to b5dff6f Compare November 11, 2025 02:54

sever-sever self-requested a review November 11, 2025 02:59

sever-sever reviewed Nov 11, 2025

View reviewed changes

src/conf_mode/system_watchdog.py Outdated Show resolved Hide resolved

Firefishy force-pushed the T7101 branch from b5dff6f to 51ae662 Compare November 11, 2025 03:19

Firefishy requested a review from Copilot November 11, 2025 03:20

Copilot started reviewing on behalf of Firefishy November 11, 2025 03:20 View session

Copilot finished reviewing on behalf of Firefishy November 11, 2025 03:22

Copilot AI reviewed Nov 11, 2025

View reviewed changes

src/conf_mode/system_watchdog.py Outdated Show resolved Hide resolved

interface-definitions/system_watchdog.xml.in Outdated Show resolved Hide resolved

sarthurdev requested changes Nov 11, 2025

View reviewed changes

interface-definitions/system_watchdog.xml.in Outdated Show resolved Hide resolved

Firefishy force-pushed the T7101 branch from 51ae662 to 587964e Compare November 11, 2025 13:03

Firefishy requested a review from Copilot November 11, 2025 13:05

Copilot started reviewing on behalf of Firefishy November 11, 2025 13:05 View session

Copilot finished reviewing on behalf of Firefishy November 11, 2025 13:08

Copilot AI reviewed Nov 11, 2025

View reviewed changes

src/conf_mode/system_watchdog.py Outdated Show resolved Hide resolved

src/conf_mode/system_watchdog.py Show resolved Hide resolved

src/conf_mode/system_watchdog.py Show resolved Hide resolved

Firefishy mentioned this pull request Nov 11, 2025

T7101: Add hardware watchdog support via systemd vyos/vyos-documentation#1703

Open

1 task

Firefishy force-pushed the T7101 branch 2 times, most recently from 4dfe354 to abf012c Compare November 11, 2025 14:23

sever-sever reviewed Nov 11, 2025

View reviewed changes

interface-definitions/system_watchdog.xml.in Outdated Show resolved Hide resolved

Firefishy force-pushed the T7101 branch from abf012c to fb03484 Compare November 11, 2025 15:07

sarthurdev requested changes Nov 11, 2025

View reviewed changes

interface-definitions/system_watchdog.xml.in Show resolved Hide resolved

Firefishy force-pushed the T7101 branch from fb03484 to 7842362 Compare November 11, 2025 20:27

Firefishy requested review from indrajitr, sarthurdev and sever-sever November 11, 2025 20:45

Firefishy requested a review from Copilot November 12, 2025 17:37

Copilot started reviewing on behalf of Firefishy November 12, 2025 17:37 View session

Copilot finished reviewing on behalf of Firefishy November 12, 2025 17:38

Copilot AI reviewed Nov 12, 2025

View reviewed changes

T7101: Add support for hardware watchdog support via systemd

811bee9

Firefishy force-pushed the T7101 branch from 7842362 to 811bee9 Compare November 12, 2025 18:09

dmbaturin requested changes Nov 13, 2025

View reviewed changes

sarthurdev requested changes Nov 13, 2025

View reviewed changes

T7101: Add hardware watchdog support via systemd #4843

Are you sure you want to change the base?

T7101: Add hardware watchdog support via systemd #4843

Conversation

Firefishy commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change summary

Types of changes

Related Task(s)

Related PR(s)

How to test / Smoketest result

Checklist:

Uh oh!

github-actions bot commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Firefishy commented Nov 10, 2025

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sever-sever commented Nov 11, 2025

Uh oh!

Uh oh!

Firefishy commented Nov 11, 2025

Uh oh!

Uh oh!

Firefishy commented Nov 12, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

Firefishy Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

Firefishy Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 12, 2025

Details

Uh oh!

dmbaturin left a comment

Choose a reason for hiding this comment

Uh oh!

dmbaturin Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Firefishy commented Nov 10, 2025 •

edited

Loading

github-actions bot commented Nov 10, 2025 •

edited

Loading